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ABSTRACT 

The Software Engineering Laboratory (SEL) Is an organization 
created nearly 10 years ago for the purpose of Identifying* 
measuring and applying quality software engineering techniques 
In a production environment (Reference 1). The members of the 
SEL include NASA/GSFC (the sponsor and organizer)* University of 
Maryland* and Computer Sciences Corporation. Since its inception 
the SEL has conducted numerous experiments, and has evaluated a 
wide range of software technologies. This paper describes 
several of the more recent experiments as well as some of the 
general conclusions to which the SEL has arrived. 

1.0 Background (Chart 1) 

Over the past 9 years* the SEL has conducted studies In 4 major 
areas of software technology: 

1. Software Tools and Environments 

2. Development Methods 

3. Measures and Profiles 

4. Software Models 

Most of these studies have been conducted by utilizing specific 
approaches* tools or models to production software problems within 
the flight dynamics environment at Goddard. By extracting 
detailed information pertaining to the problem* environment* 
process and product* the SEL has been able to gain some Insight 
into the relative impact that the various technologies may have 
on the quality of the software being developed. 

More detailed descriptions of the overall measurement process as 
well as the SEL studies may be found in References 1, 2* and 3. 

This brief paper will describe some of the more recent* specific 
experiments that have been conducted by/in the SEL and just what 
types of insight may be provided In areas of: 

1. Tools and Environments 

2. Software Testing 

3. Design Measures 

4. General Trends 

*The work described in this paper has been extracted from reports and studies carried 
out by members of the SEL. 
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TYPE OF 
SOFTWARE: 


SCIENTIFIC# GROUND-BASED# INTERACTIVE GRAPHIC, 
MODERATE RELIABILITY AND RESPONSE REQUIREMENTS 


LANGUAGES: 85% FORTRAN, 15% ASSEMBLER MACROS 


COMPUTERS: IBM MAINFRAMES# BATCH WITH TSO 


PROJECT CHARACTERISTICS: AVERAGE HIGH 


LOW 


DURATION (MONTHS) 


16 21 13 


EFFORT (STAFF-YEARS) 


8 24 


2 


SIZE (1000 LOC) 
DEVELOPED 
DELIVERED 


57 142 22 

62 159 33 


STAFF (FULL-TIME 
EQUIVALENT) 
AVERAGE 
PEAK 

INDIVUALS 


5 11 2 

10 24 4 

14 29 7 


APPLICATION EXPERIENCE 
(YEARS) 

MANAGERS 6 7 

TECHNICAL STAFF 4 5 


5 

3 


OVERALL EXPERIENCE 
(YEARS) 

MANAGERS 
TECHNICAL STAFF 


10 14 8 

9 11 7 


FIGURE 1. FLIGHT DYNAMICS SOFTWARE 
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The Flight Dynamics environment typically is a FORTRAN environ- 
ment building software systems ranging in size from 10*000 to 
150*000 lines of code - (see Figure 1). 

2.0 Software Tools/Environments* (Chart 2 and Reference 4) 

One of the more interesting studies that was conducted within the 
past several years* was one in which an attempt was made to 
measure the impact of several development approaches (related to 
environment support) on the quality of software within the flight 
dynamics discipline. 

The three points of study include: 

1 . Software Tool s 

2. Computer Support 

3. Number of Terminals/Programmer 

The quality of the product was measured using 4 attributes 
1 ncl ud 1 ng : 

1. Productivity - Number of developed lines of code per man 
month . 

2. Reliability - Number of errors reported per 1*000 lines 
of code. 

3. Effort to Change - (Average number of man hours 
required to make a software modification). 

4. Effort to Repair (Average number of man hours required to 
correct an identified error) 

2.1 Experiment Description (Chart 3) 

In carrying out the study* a review of all projects for which 
detailed project history data was available and complete was 
undertaken. From the completed 50 projects* 14 were selected 
because of the quality and completeness of the relevant data and 
more important! y because of the general similarity of 
complexity of problems that the software was attempting to solve. 

Fourteen projects ranging in size from 11*000 lines of code to 
136*000 lines of code were selected. These projects had 
information describing the environment under which they were 
developed and additional information such as the number and 
quality of automated tools utilized and the number of interactive 
terminals available to the programming staff. 


*Lead investigators of this work included F. McGarry and J. Valett of NASA/GSFC 
and D. Hall of NASA/HQ. 
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The 14 projects selected all dealt with tasks in solving attitude 
determination and control related problems. The projects were 
completed between the years 1978 to 1984. 

The projects also had detailed information as to manhours* size, 
error history* and effort required to make all changes and 
corrections to the software. 

2.2 Project Variations (Chart 4) 

In attempting to characterize each of the development projects* 
a ranking scheme was used for this particular study. It was 
found that the availability of terminals ranged from a low of 
less than 1 per 8 programmers to a high of better than 1 per 2 
programmers. 

There were a total of 21 tools considered in this study that 
were applied by at least some of the projects studied. Such 
tools as documentation aids* preprocessors* test generators and 
program optimizers were among the tools considered. 

It was also found that the distribution of level of use for tools 
ranged from a low of only 1 or 2 automated tools being used* to a 
high of more than 8 automated tool s being used. These tool s al so 
were rated as far as the actual usage by the particular project 
and also there was a rating for each tool of the assessed 
’quality’ of the particular tool. Quality here was rated for 
each tool on a scale of 1 to 5 and was a subjective rating 
determined by the software manager. 

There were a total of 11 characteristics that made up the 
computer support measure. These 11 Included: 

o Terminal Accessibility o Offline Storage 

o Turn around time o Interactive Availability 

o Compiler Speed o Terminals/programmers 

o System Reliability (2 measures) o Avg. CPU Utilization 

oDIrect Storage o A c c esslbil Ity of all 

resources 


2.3 Study Results (Chart 5) 

The results of this particular study were encouraging on the one 
hand and quite perplexing on the other. 

2.3.1 Tool usage results showed that as the number and quality 
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of automated tools Increased, there were significant Increases in 
3 of the 4 quality measures used in this study: 

1. Productivity Increased as tool usage increased 

2. Maintainability (effort to change/effort to repair) 
Improved as the number and quality of tools increased. 

3. Reliability did not seem to be significantly Impacted in 
this one particular study. 

2.3.2 Computer Environment 

Although all of the experimenters felt that there would be 
significant increases in all quality measures as the overall 
quality of computer support increased, none of the measures 
proved to be significant for this one particular study. It could 
not be shown that an improved computer support environment (at 
leastas far as the way the SEL described support environment) 
directly, favorably Impacted the four quality measures used by 
the SEL. 

This particular study is still undergoing further analysis. 

2.3.3 Terminal Usage 

The most perplexing result of this experiment study was the 
one in which the SEL attempted to assess the Impact that 
Increased number of terminals would have on the four measures 
described. 

Although the experimenters expected to observe an Increase in 
both productivity and software reliability as the number of 
terminals made available Increased, the study found just the 
opposite. Both productivity and reliability of software 
decreased as the ratio of terminals available Increased. There 
was no significance in the results for maintainability (effort to 
change/effort for repair). 

Numerous suggestions have been put forth in attempting to explain 
this phenomena. Some felt that the increased terminal usage 
possibly was not properly accompanied with interactive support 
tools in the particular environment. 

Another idea was that the increased terminal availability without 
proper training for the programmers led to a less disciplined 
approach by the programmers. 
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There are several other possible explanations of the results and 
for that reason, this particular study has been continuing and 
will be attempting to more thoroughly analyze this data as well 
as the additional projects that have been completed in this 
env i ronment . 

3.0 Software Testing 

A second general set of studies that has been conducted over the 
past several years within the SEL has been directed toward gaining 
insight into approaches to testing software. Since this phase of 
the development life cycle had previously been determined to 
consume at least 30 percent of the development resources 
(Reference 5), it was deemed as a critically important discipline 
to study. Two major experiments were conducted during 1984 and 
1985 in an attempt to: 

1. Determine the overall coverage of software in the 
typical testing scenario utilized in the flight dynamics 
software development. 

2. Investigate the relative merits of three standard 
testing approaches: 

o functional testing 
o structural testing 
o code reading 


3.1 Test Coverage* (Chart 6 and Reference 6) 

The first experiment on testing was designed to determine the 
extent to which typical testing techniques within the flight 
dynamics environment amply exercised the software that had been 
built. This particular environment utilizes functional testing 
during both the system test phase as well as the acceptance test 
phase. 

By instrumenting a major flight dynamics system, then by 
executing the series of both system tests and acceptance tests - 
experimenters could first determine the coverage attained in the 
test phases. Next, the experimenters monitored the operational 
execution of this same software over a period of months to 
determine the extent to which portions of the completed software 
were utilized. Finally, the experimenters analyzed uncovered 
errors in an attempt to determine if the errors occurred in 
portions of the system that had not been exercised during the 


*The lead investigator for this work was Jim Ramsey of Univ. of MD 
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test phase of development. The software studied was a major 
subsystem of a mission planning tool and consisted of 68 modules 
(Fortran subroutines) with 10,000 lines of code. There were 10 
functional tests making up the acceptance test plan for the 
subsystem and during the operational phase, the experimenters 
monitored 60 operational execution of the software. 

3.1.1 Test Coverage Results (Chart 7) 

The managers of the flight dynamics development systems noted 
that the approach to testing had historically been quite good 
(relatively few errors found In operations) and they expected 
that the coverage found for this one experiment would be quite 
high (few modules would be not executed). The results of the 
experiment showed that for the 10 functional tests executed, only 
75 percent of the 68 modules were executed and less than 60 
percent of the total executable code was covered in the tests. 


Additionally, the series of operational executions showed that a 
slightly higher percentage of both number of modules and lines of 
code were executed for this series of 60 executions. 

Finally, all of the error reports were reviewed to determine in 
which portion of the system the errors had occurred. It was 
found that 8 errors had been recorded during the extended 
operational phase of the software, but it was found that none of 
the reported errors occurred In software that had not been 
executed during the acceptance test phase. 

This Initial study seemed to Indicate that the functional testing 
approach was properly leading to correct portions of the system 
being executed and it also was very representative of the 
operational usage of the software. 

The results of this study indicated that further investigations 
into the various approaches to testing may be worthwhile to 
determine just which approaches were most effective In uncovering 
errors in the software Itself. 

3.2 Software Testing Techniques (Chart 8 and Reference 7) 

Another study was conducted where three programs were seeded with 
a number of faults and 32 professional programmers from NASA/GSFC 
and from Computer Sciences Corporation (CSC) participated In an 
experiment to determine which techniques were effective in 
uncovering these faults. 

The three testing approaches included: 


*The lead investigator for this study was Rick Selby of Uni v. of MD 
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o Functional Testing 
o Structural Testing 
o Code Reading 

All programmers participated in applying each of the three 
techniques. 

When performing functional tests, the programmers were required 
to use the functional requirements along with test results to 
isolate faults - they were not to look at the source code itself 
until after testing was completed. 

Those programmers performing structural testing used the source 
code and test results but did not use the functional 
requ i rements . 

Code reading was carried out with no executions of the software. 
Those performing code reading reviewed the requirements and also 
looked at the source code. 

3.2.1 Testing Technique Results (Charts 9 and 10) 

The. results of this experiment indicated that code reading is the 
most effective of the three testing techniques studied. This 
technique uncovered an average of 61 percent of all seeded faults 
while functional testing uncovered 51 percent and structural 
testing uncovered 38 percent. 


Before the test, most of the managers in the SEL felt that code 
reading would prove to be a very effective testing technique, 
although they also felt that it would probably be the most costly 
in manhours to apply; but the results of the experiment indicated 
that code reading also was the most cost effective technique (3.3 
faults per manhour vs 1.8 faults per manhour for structural and 
for functional testing). It was also noteworthy that, before the 
experiment, less than 1 out of 4 persons participating in the 
experiment predicted that code reading would be the most 
effective approach. 

An additional observation that was made after the testing results 
were compiled was that there seemed to be a difference In the 
relative effectiveness of each of the testing approaches as the 
size of the software being tested Increased. For the smaller 
program, code reading was by far the most effective technique, 
but for the larger program, functional testing seemed to be quite 
effective. This observation may indicate that there should be a 
size limit on how much code is utilized in a code reading 
exercise. Further tests are planned for these studies. 
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4,0 Software Measures 


Over the past 6 to 8 years, the SEL has defined, studied, and 
evaluated numerous measures applicable to software development 
and management (References 8, 9, 10). Most of these measures 
have focused on one phase of the software life cycle - the code/ 
unit test phase. In an attempt to define and apply measures in 
earlier phases of the life cycle, the SEL has been reviewing 
several approaches to qualifying or measuring aspects of the 
software during the specifications phase and during the design 
phase. Work on the specification phase was reported at the Ninth 
Software Engineering Workshop and may be found 1ft reference 11 
and 12. One additional piece of work that has been conducted for 
the design phase will be discussed here. 

4.1 Software Design Measures* (Charts 11 and 12 Reference 
13, 14) 

In an attempt to qua! ify software designs, a study was conducted 
to determine if module strength may be utilized as a guideline 
for software mod u 1 ar 1 zat i c n. Although the definitions of 
strength may be wel 1 understood, the parameter may not be easy to 
determine based solely on a structure chart or data flow diagram 
which may be produced during the design phase of software 
devel opment . 

For the purposes of this study, strength is defined as the 
’singleness of purpose’ that a software module inherently 
contains. Singleness of purpose is a subjective parameter 
assigned at design time by the developer/manager. From a list of 
potential functionality that a component may have (e.g. computa- 
tional, control, data processing, etc.) the programmer determines 
which functions that module contains. High strength would be 
attributed to those components which have but a single function 
to perform, medium to 2 and low strength would have three of more 
functions to perform. 

The study examined 450 Fortran modules (from 4 systems) which 
were built by approximately 20 different developers. 

Typical SEL data, which includes detailed cost and error data for 
all modules was also available for all of the modules. The 450 
modules used for this study had a fairly even distribution in 
size as well as in design strength. Small modules (104 of the 
450) were those with up to 31 executable statements, medium (148 
of 450) were those with up to 64 executable statements and there 
were 151 large modules which had more than 64 executable 
statements . 


*The lead investigators for this study were D. Card and G. Page of CSC and 
F. Me Garry of NASA/GSFC 
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The objective of the study was to determine if strength of 
modules as determined at design time was related to the cost and 
reliability of the completed product. 

4.2 Results of the Study on Strength (Charts 13, 14, 15) 

The results of the study in the SEL indicated that module 
strength is indeed a reasonable criteria for defining software 
modularization. When examining the reliability of the 450 
modules, it was found that 50 percent of the high strength 
modules had zero defects while for medium strength modules 36 
percent had zero defects and low strength modules only 18 percent 
of the modules had zero defects. Similar trends were found for 
the modules of medium error proneness (up to 3 errors per 1000 
lines of code) and for modules having a high error rate (over 3 
errors per 1000 lines of code). 

The distribution of the ’buggy’ modules (over 3 errors per 1000 
lines of code) was shown to tend more toward low strength as 
opposed to high strength. Forty- four percent of the buggy 
modules had low strength while only 20 percent of the buggy 
modules were found to have high strength. 

Several additional observations were made while conducting this 
particular study. When the characteristics of the individual 
programmers were reviewed, it was found that those programmers 
who produced high quality software (low error rate and high 
productivity) tended to design modules of high strength but they 
also did not show a preference for writing modules of any 
specific size. Good programmers generated modules of size that 
seemed to best suit their design and they did not artificially 
constrain themselves to writing small modules. 

5.0 General Trends and Observations 

Over the past several years, the SEL has conducted numerous 
studies and experiments in an attempt to better understand the 
impact that various software techniques may have on producing 
improved software. In addition to the specific studies conducted 
such as the ones briefly discussed in sections 2, 3, and 4, the 
SEL has observed general trends in the development and 
measurement of software. The observations include such points as 
trends in software reuse, trends in utilization of improved 
software development technology, and the overall impact of 
improved developed techniques in the cost and reliability of 
software over a long period of observation time. Some of these 
general observations are summarized here. 
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5.1 Trends in Computer Use and Technology Application (Charts 
16, 17) 

From data that has been collected on nearly 60 projects over the 
past 9 years, one trend that has been noted is the tendency to 
make heavier and heavier usage of available computer support. In 
1977 and 1978, computer use averaged approximately 100 runs per 
1000 lines of developed source code while in 1982 and 1983 the 
average use increased to nearly 250 runs per 1000 lines of 
source. This trend continues to increase within the flight 
dynamics environment being studied. 

Simultaneously, it was noted that the use of more and more 
structured development practices, improved management approaches 
and overall higher quality software engineering has continually 
increased. Each project has been rated on its application of 
over 200 software techniques (see reference 15) in an attempt to 
quantify the overall level of development and management tech- 
nology util ized for a project. The aggregate of the total set of 
techniques applied results in a rating termed the Software Tech- 
nology Index. From an average index of less than 100 in 1976 to 
1978, it was found that the overall development techniques have 
increased to an average of over 140 in the 1980's. This seems to 
point to improved training, better discipline, improved access to 
tools and possibly better informed management practices. 

Although both parameters (computer use and software technology 
index) seemed to generally Increase over the past 7 or 8 years, 
there is no observed correlation between these two factors. 

5.2 Trends in Software Reuse (Chart 18) 

Another general observation that was made from the detailed 
development data collected by the SEL, was that the reuse of 
software has shown general trends of Increase. Typical software 
systems in the years 1 977 to 1 979 averaged about 15 or 20 percent 
reused code while in the 1982 to 1984 timeframe the average reuse 
has increased to 30 to 35 percent. 

Although this reuse is certainly tending in the right direction, 
the SEL has not conducted detailed studies to determine what the 
driving factors are in improving the percentage of reuse. The 
trends are probably indicative of Improvements in design 
technique as well as numerous other factors, but studies have 
just recently been initiated in the SEL to determine how the 
trend can be improved at a even faster pace. 

It has also been observed in the SEL data that there does not 
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seem to be a direct relationship between projects that are rated 
as having a high software technology index and having a high rate 
of software reuse. But this may not be a surprise since one 
would expect that high technology usage would lead to follow on 
systems being able to pick up or reuse software produced by the 
projects using disciplined approaches for development and 
management . 

5.3 Impact of Development Technologies (Chart 19) 

Probably the most basic goal that the SEL has, is to determine 
the impact that specified software development / management 
techniques have on the cost and reliability of software. With 
nearly 60 projects having been closely monitored over the past 8 
or 9 years»the SEL attempted to look at general trends inthe 
reliability and cost of these projects as measured against the 
software technology index computed for each of these projects. 
The 200 parameters factored into this index represent everything 
from structured techniques to disciplined management approaches 
to configuration control procedures. It is one attempt to 
characterize each of the projects with a single value. 

This technology index correlates very w e 1 1 ( r = . 8 2 ) w i t h 
reliability of software in the SEL. Those projects with a higher 
rating of good development practices were the projects with the 
lower fault rates of the product. 

Unfortunately, the Impact of this technology Index on 
productivity is quite unclear. The first general observation 
that has been made is that there is not a clear favorable impact 
on development cost (cost per line of code) with projects with 
higher values of this technology index. Studies are continuing 
in an attempt to more objectively compute this technology rating 
so that a more conclusive statement can be made. Some 
researchers also have suggested that it is not to be unexpected 
that the specific development cost may not decrease but since 
the reliability has improved and the overall software structure 
has Improved, the maintenance activity will be the beneficiary of 
the overall cost savings, not the development cost. 

5.4 Can Software Technology be Measured? (Chart 20 and Reference 
3) 

Another major question that software engineers address is whether 
or not software technology can be measured at all. By utilizing 
reliability as one major aspect of software quality, the SEL 
attempted to determine to what extent software development/ 
management practices could be measured. 
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There are three levels of development practices which the SEL has 
hoped and attempted to measure. First* there are Individual 
specific techniques such as the use of structured code or chief 
programmer team or the use of PDL in design* etc. 

Second* there Is the usage of a software methodology which is a 
combination of several methods Into a single disciplined 
approach. This could, be the set of methods known as structured 
techniques which reflect the use of 6 or 8 individual practices 
such as top down development* structured code* code reading and 
usage of Unit Development Folders (UDF). 

Finally* the attempt has been made to measure the Impact of the 
total technology Index which encompasses all disciplined 
management/development practices. This signifies the level to 
which the project has attempted to apply recommended software 
development techniques. 

The results of this study indicated; 

1. An individual technique cannot be effectively measured in 
a production environment such as the one In which the SEL Is 
conducting studies, (r = .37 Is a typical value found In 
correlating PDL usage and reliability), 

2. Disciplined methodologies (combining techniques into a 
single disciplined approach) can be measured (r = .65 for one 
particular study) and the approaches called Modern Programming 
Practices (6 techniques) has a significant* measurable* favorable 
impact on software reliability. 

3. Total Software Technology can be measured (r = .82 for 
this one study) and higher levels of applied technology have a 
marked favorable impact on the reliability of software. 

The trends and observations noted here are based on approximately 
8 years of data collection and experimentation within the SEL. 
Approximately 55 projects have been studied and the research is 
continuing and will continue In the future. 

Many of the results are Inclusive* but with each experience and 
study* greater insight is provided into the overall 
characteristics of the software development process. 
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STUDIES AND EXPERIMENTS 
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MEASURING THE EFFECTS OF ENVIRONMENT 
ON SOFTWARE DEVELOPMENT 
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ENVIRONMENT VARIATIONS 
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TEST COVERAGE 
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TEST COVERAGE RESULTS 
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STUDIES OF SOFTWARE 
TESTING TECHNIQUES 
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Code Functional Structural Code Functional Structural 

Reading Testing Testing Reading Testing Testing 


Code Reading Proved To Be the Best Technique in Terms of the Total Number 
of Faults Detected and the Faults Detected Per Hour of Effort 

Prior To the Experiment Only 23% of the Subjects Believed Code Reading To 
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SOFTWARE DESIGN MEASURES 
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FAULT RATE FOR CLASSES 
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DESIGN MEASURES SUMMARY 
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COMPUTER USE AND TECHNOLOGY 

TIME TRENDS 
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EFFECT OF TECHNOLOGY ON 
COMPUTER USE 
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TRENDS IN SOFTWARE REUSE 
(BASED ON 15 PROJECTS OF 
SIMILAR CHARACTERISTICS) 
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EFFECTS OF DEVELOPMENT 
TECHNOLOGIES 
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EFFECT OF TECHNOLOGY USE ON 
SOFTWARE RELIABILITY 
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